<html>
 <head>
  <meta charset="UTF-8">
 </head>
 <body>
  <h3 data-lake-id="gl73M" id="gl73M"><span data-lake-id="uc53562d7" id="uc53562d7">问题发现</span></h3>
  <p data-lake-id="u6dd0d548" id="u6dd0d548"><br></p>
  <p data-lake-id="u5f35d316" id="u5f35d316"><span data-lake-id="u770726e7" id="u770726e7">线上兼容系统报警，提示有频繁的FullGC以及GC耗时问题比较严重。</span></p>
  <p data-lake-id="ufe9855d0" id="ufe9855d0"><span data-lake-id="u91d19c09" id="u91d19c09">​</span><br></p>
  <p data-lake-id="ud12ee63b" id="ud12ee63b"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702880427123-d4534019-4b0f-470a-995b-9c2c8b42d2fb.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_21%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u1c280969" id="u1c280969"><br></p>
  <h3 data-lake-id="zKtIL" id="zKtIL"><span data-lake-id="ud68f72ec" id="ud68f72ec">问题定位</span></h3>
  <p data-lake-id="u97b1f6e1" id="u97b1f6e1"><br></p>
  <p data-lake-id="uf036ea00" id="uf036ea00"><span data-lake-id="u85c5901a" id="u85c5901a">在收到FullGC报警之后，登录到内部的监控系统，看一下集群整体的GC情况（如果没有这样的监控系统，可以去机器上查看GC日志）：</span></p>
  <p data-lake-id="uff3fe2f3" id="uff3fe2f3"><span data-lake-id="ub5a10a5c" id="ub5a10a5c">​</span><br></p>
  <p data-lake-id="u3cc0686d" id="u3cc0686d"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702879034460-ec898ac3-abed-4ae2-acf7-13777385b45c.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_33%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="ued10c547" id="ued10c547"><br></p>
  <p data-lake-id="u235cf071" id="u235cf071"><span data-lake-id="ue87e938d" id="ue87e938d">可以看到，集群的GC次数是3小时内有十几次了，那么去看一下单机的情况。</span></p>
  <p data-lake-id="u478b9aa4" id="u478b9aa4"><span data-lake-id="uebed87bb" id="uebed87bb">​</span><br></p>
  <p data-lake-id="u79b1ecb1" id="u79b1ecb1"><span data-lake-id="u315d61d1" id="u315d61d1">这里我先去看了一下报警的那台机器，然后有随机挑了几台线上机器去查看，看到一个比较特殊的现象：</span></p>
  <p data-lake-id="uc3cc1dec" id="uc3cc1dec"><span data-lake-id="ua8d8fe37" id="ua8d8fe37">​</span><br></p>
  <p data-lake-id="uaade750d" id="uaade750d"><span data-lake-id="u79a2d15c" id="u79a2d15c">那就是并不是所有机器都存在FullGC的情况，有些机器的堆内存的水位甚至还挺低的。但是这里没多想，接着去看堆dump去了。</span></p>
  <p data-lake-id="ue9c6e95b" id="ue9c6e95b"><span data-lake-id="ud5b883dd" id="ud5b883dd">​</span><br></p>
  <p data-lake-id="uc1fadb6a" id="uc1fadb6a"><span data-lake-id="u9546e5d8" id="u9546e5d8">我们因为内部有进行堆dump以及分析的工具，如果么有的话可以使用jmap或者arthas获取堆dump。然后再使用Java VisualVM、</span><span data-lake-id="u86b7229d" id="u86b7229d" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">Memory Analyzer Tool等工具进行分析。</span></p>
  <p data-lake-id="u1bcb8df8" id="u1bcb8df8"><span data-lake-id="u23bddb0c" id="u23bddb0c" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">​</span><br></p>
  <p data-lake-id="uf301be02" id="uf301be02"><span data-lake-id="uf535aff5" id="uf535aff5" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">我这里是分别对存在FullGC的机器、当前堆内存占用比较高的机器、以及内存占用并不高也没有频繁GC的机器进行了dump。</span></p>
  <p data-lake-id="uf72f03a1" id="uf72f03a1"><span data-lake-id="udb9eb7b4" id="udb9eb7b4" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">​</span><br></p>
  <p data-lake-id="u746cf0ba" id="u746cf0ba"><strong><span data-lake-id="u2182de01" id="u2182de01" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">之所有多次dump，主要是为了作对比。</span></strong></p>
  <p data-lake-id="u2114b9ca" id="u2114b9ca"><strong><span data-lake-id="u2007d8b3" id="u2007d8b3" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">​</span></strong><br></p>
  <p data-lake-id="u62e30e8d" id="u62e30e8d"><span data-lake-id="ub9f1792e" id="ub9f1792e" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">在分析过程中，发现堆内存占用比较高的机器和FullGC比较频繁的机器中，存在着一些比较特殊的现象。</span></p>
  <p data-lake-id="ua74c9169" id="ua74c9169"><span data-lake-id="u122afd36" id="u122afd36" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">​</span><br></p>
  <p data-lake-id="ue27e9849" id="ue27e9849"><span data-lake-id="u7aba01c0" id="u7aba01c0" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">首先是有大对象占了2个多G的内存。</span></p>
  <p data-lake-id="u0256c770" id="u0256c770"><span data-lake-id="u1ab4dca7" id="u1ab4dca7" class="lake-fontsize-12" style="color: rgb(36, 41, 46)">​</span><br></p>
  <p data-lake-id="ufa9213c6" id="ufa9213c6"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702879386755-26739c8b-7cc8-4b2d-987d-da1dd6940342.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_50%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u188c0b39" id="u188c0b39"><br></p>
  <p data-lake-id="ub44f8519" id="ub44f8519"><span data-lake-id="u5f8d99c9" id="u5f8d99c9">然后再进一步查看大对象内容：</span></p>
  <p data-lake-id="u64c3f5d4" id="u64c3f5d4"><span data-lake-id="ued7bd133" id="ued7bd133">​</span><br></p>
  <p data-lake-id="u46c35e75" id="u46c35e75"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702879491271-715cd913-fe9b-4a3d-9ea0-13013ebeba1a.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_50%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="ub28b6c2e" id="ub28b6c2e"><br></p>
  <p data-lake-id="ud93bd349" id="ud93bd349"><span data-lake-id="ub210c6cd" id="ub210c6cd">发现有一个ArrayList中存放了60多万个CollectionCaseDO对象。</span></p>
  <p data-lake-id="uf710dc77" id="uf710dc77"><span data-lake-id="uff0a547a" id="uff0a547a">​</span><br></p>
  <p data-lake-id="uf9962543" id="uf9962543"><span data-lake-id="u10907b6d" id="u10907b6d">这个CollectionCaseDO对象我就比较熟悉了，是我们自己的业务模型。但是竟然会在内存中加载这么多就很奇怪了。</span></p>
  <p data-lake-id="u464845a5" id="u464845a5"><span data-lake-id="u5a5048ff" id="u5a5048ff">​</span><br></p>
  <p data-lake-id="u4b7a9417" id="u4b7a9417"><span data-lake-id="u00d2794c" id="u00d2794c">看到这里我有两个猜测：</span></p>
  <p data-lake-id="u7179d07f" id="u7179d07f"><span data-lake-id="uff54781a" id="uff54781a">​</span><br></p>
  <p data-lake-id="u84833895" id="u84833895"><span data-lake-id="uad0e8f16" id="uad0e8f16">1、在一个bean中有一个List&lt;CollectionCaseDO&gt;的成员变量，在代码中会多次向其中add，导致他有这么大的量。</span></p>
  <p data-lake-id="u1c975452" id="u1c975452"><span data-lake-id="u9b91c4ee" id="u9b91c4ee">2、在代码中有一个地方在做查询的时候没有做好条件过滤及分页，导致数据库查询了大量数据。</span></p>
  <p data-lake-id="u34ddedd7" id="u34ddedd7"><span data-lake-id="u9e394220" id="u9e394220">​</span><br></p>
  <p data-lake-id="u243dac04" id="u243dac04"><span data-lake-id="u08cb8cd7" id="u08cb8cd7">第一个猜测很快被我排除了，因为我去全局搜索了代码，并没有发现这种用法。那么就只剩第二种了，接下来排查在哪里出现的这个问题查询。</span></p>
  <p data-lake-id="uabfbc945" id="uabfbc945"><span data-lake-id="u68ab325e" id="u68ab325e">​</span><br></p>
  <p data-lake-id="ue3e43469" id="ue3e43469"><span data-lake-id="ub177c411" id="ub177c411">然后我又想到，线上不是所有机器都有这个现象，只有部分机器，并且通过监控发现，出问题的机器堆内存是逐步增长起来的：</span></p>
  <p data-lake-id="u9dbcfd46" id="u9dbcfd46"><span data-lake-id="u6f855f34" id="u6f855f34">​</span><br></p>
  <p data-lake-id="ufafc9cb0" id="ufafc9cb0"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702879718972-49d2feb6-48f6-41d9-b2fa-96edb3449a93.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_22%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u7acc084c" id="u7acc084c"><br></p>
  <p data-lake-id="u8797afdc" id="u8797afdc"><span data-lake-id="uaa811e51" id="uaa811e51">于是，根据问题发生的时间点，去查日志。</span></p>
  <p data-lake-id="u253eb416" id="u253eb416"><span data-lake-id="u6963e6fe" id="u6963e6fe">​</span><br></p>
  <p data-lake-id="ucc9be4c3" id="ucc9be4c3"><span data-lake-id="ua21d7b38" id="ua21d7b38">在查日志的之前，我根据上面的情况，以及dump的信息，进一步定位到这个问题应该和我们的一个查询接口有关。</span></p>
  <p data-lake-id="u8bbb151c" id="u8bbb151c"><span data-lake-id="u49b30863" id="u49b30863">​</span><br></p>
  <p data-lake-id="u812f8f16" id="u812f8f16"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702879979493-fc330314-721b-4f40-bc74-a6e3af642b8a.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_36%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u5c04d4ff" id="u5c04d4ff"><span data-lake-id="ue2688192" id="ue2688192">​</span><br></p>
  <p data-lake-id="u5869f06e" id="u5869f06e"><span data-lake-id="u6bc63dfd" id="u6bc63dfd">于是通过这个接口的关键日志进行查询，还真的让我查到了端倪。</span></p>
  <p data-lake-id="ucab17770" id="ucab17770"><span data-lake-id="ub4ce7c5a" id="ub4ce7c5a">​</span><br></p>
  <p data-lake-id="u7f24d601" id="u7f24d601"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702880468284-890e1eb5-f25e-4c21-9843-4001383d0a6c.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_94%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u6af5fb1b" id="u6af5fb1b"><br></p>
  <p data-lake-id="u473871de" id="u473871de"><span data-lake-id="u5aad112e" id="u5aad112e">在内存两次增长的时间点，刚好有两条特殊的日志。</span></p>
  <p data-lake-id="ub0f1e088" id="ub0f1e088"><span data-lake-id="uae6c4816" id="uae6c4816">​</span><br></p>
  <p data-lake-id="ue595f8a8" id="ue595f8a8"><span data-lake-id="ue4869d87" id="ue4869d87">正常的查询，参数中是要带一个查询的id或者当前的坐席的，如：</span></p>
  <p data-lake-id="u22ee58b1" id="u22ee58b1"><span data-lake-id="uc0430eb3" id="uc0430eb3">​</span><br></p>
  <p data-lake-id="u1d3cd59f" id="u1d3cd59f"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702880138834-76381c2f-b580-4e4b-b80e-a76d27fe98a8.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_53%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u1cb5e60f" id="u1cb5e60f"><span data-lake-id="u12367089" id="u12367089">​</span><br></p>
  <p data-lake-id="u36a14a38" id="u36a14a38"><span data-lake-id="ub5fecf2a" id="ub5fecf2a">但是上面的问题查询没有带这个ID，那么看了一下代码，这是一个根据ID查询详情的接口，但是发现同事的代码中并没有对这个caseId做非空校验，然后在用户未传递caseId的时候，用了个queryList，就会把所有的案件都查出来放到List中。。。。</span></p>
  <p data-lake-id="u10c8c009" id="u10c8c009"><span data-lake-id="u3ddfc4ac" id="u3ddfc4ac">​</span><br></p>
  <p data-lake-id="u1d357470" id="u1d357470"><img src="https://cdn.nlark.com/yuque/0/2023/png/5378072/1702880259932-ce1a3b43-accd-45c5-8c6b-a749c0cbb969.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_29%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u625917f0" id="u625917f0"><span data-lake-id="u646736de" id="u646736de">​</span><br></p>
  <p data-lake-id="u65f99e69" id="u65f99e69"><span data-lake-id="u83aafa64" id="u83aafa64">截止到这里，后端的问题基本上定位到了，因为没有传ID，并没有做校验，导致一次查询把所有数据都查出来，放到了List中，然后导致大对象被放到老年代占用了大量空间，因为有多次查询，导致FullGC多次。</span></p>
  <p data-lake-id="ufc09cbd7" id="ufc09cbd7"><span data-lake-id="uf114e442" id="uf114e442">​</span><br></p>
  <p data-lake-id="uc85d768d" id="uc85d768d"><span data-lake-id="u229a5930" id="u229a5930">后面为啥没传caseId就让前端检查了一下，发现是前端的bug，但是也确实是后端没做好校验导致的。</span></p>
  <h3 data-lake-id="iemXy" id="iemXy"><span data-lake-id="ue2651d04" id="ue2651d04">问题解决</span></h3>
  <p data-lake-id="u076e8726" id="u076e8726"><br></p>
  <p data-lake-id="u874b3237" id="u874b3237"><span data-lake-id="u26acfc2a" id="u26acfc2a">问题定位了，解决很简单了，就对caseID做一下非空校验就行了。如果发现没传，直接报错返回即可。</span></p>
 </body>
</html>